Papers with exponential moving average
LoRA-MGPO: Mitigating Double Descent in Low-Rank Adaptation via Momentum-Guided Perturbation Optimization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) adapts large language models by training only a small fraction of parameters, but as the rank of the low-rank matrices increases, LoRA exhibits an unstable “double descent” phenomenon, which delays convergence and impairs generalization by causing instability due to the attraction to sharp local minima. |
| Approach: | They propose a framework that incorporates Momentum-Guided Perturbation Optimization (MGPO) MGPO stabilizes training dynamics by mitigating double descent phenomenon and guiding weight perturbations using momentum vectors from the optimizer’s state. |
| Outcome: | The proposed framework improves performance on natural language understanding benchmarks and shows that it improves convergence and generalization. |
MAFiD: Moving Average Equipped Fusion-in-Decoder for Question Answering over Tabular and Textual Data (2023.findings-eacl)
Copied to clipboard
| Challenge: | Experimental results show that Transformer-based questions have a "long" hybrid sequence over tabular and textual elements, causing long-range reasoning problems. |
| Approach: | They propose a moving average-equipped fusion-in-decoder to handle long-range reasoning problems . they use FiD and EMA to combine different levels of reasoning . |
| Outcome: | Experimental results show that the proposed model increases exact matching and F1 by 1.1 and 1.7 on the blind test set. |
DemaFormer: Damped Exponential Moving Average Transformer with Energy-Based Modeling for Temporal Language Grounding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Temporal Language Grounding (TLG) is a task to determine temporal boundaries of video moments that correspond to a language query. |
| Approach: | They propose an energy-based model framework to explicitly learn moment-query distributions. |
| Outcome: | The proposed model outperforms the state-of-the-art models on four public temporal language grounding datasets. |
Noisy Pair Corrector for Dense Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing dense retrieval models assume that query-document pairs are exactly matched, resulting in mismatched-pair noise. |
| Approach: | They propose a novel approach to train an effective model with mismatched-pair noise. |
| Outcome: | The proposed model performs well on natural question and triviaQA, code-search benchmarks and SO-DS. |
Emergent Abilities of Large Language Models under Continued Pre-training for Language Adaptation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing large language models are notoriously English-centric, and their performance has been reported to drop significantly in lessresourced languages. |
| Approach: | They propose a language-agnostic benchmark for in-context learning that reveals catastrophic forgetting early on CPT when English is not included. |
| Outcome: | The proposed method does not impact validation perplexity but is critical for emergence of downstream capabilities in the target language. |
ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing efficiency methods mainly target inference via token reduction or merging, offering limited benefits during training. |
| Approach: | They propose an adaptive token pruning method that uses a teacher-student framework to prune MLLMs to reduce inference costs. |
| Outcome: | The proposed method matches the peak accuracy of standard training on MVBench up to **2 faster**, using only **38% of the tokens. |